{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rwxGnsA92emp"
      },
      "source": [
        "##### Copyright 2018 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "CPII1rGR2rF9"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JtEZ1pCPn--z"
      },
      "source": [
        "# Custom training: walkthrough"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GV1F7tVTN3Dn"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/tutorials/customization/custom_training_walkthrough\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/customization/custom_training_walkthrough.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/en/tutorials/customization/custom_training_walkthrough.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/en/tutorials/customization/custom_training_walkthrough.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LDrzLFXE8T1l"
      },
      "source": [
        "This tutorial shows you how to train a machine learning model with a custom training loop to *categorize* penguins by species. In this notebook, you use TensorFlow to accomplish the following:\n",
        "\n",
        "1. Import a dataset\n",
        "2. Build a simple linear model\n",
        "3. Train the model\n",
        "4. Evaluate the model's effectiveness\n",
        "5. Use the trained model to make predictions\n",
        "\n",
        "## TensorFlow programming\n",
        "\n",
        "This tutorial demonstrates the following TensorFlow programming tasks:\n",
        "\n",
        "* Importing data with the [TensorFlow Datasets API](https://www.tensorflow.org/datasets/overview#load_a_dataset)\n",
        "* Building models and layers with the [Keras API](https://www.tensorflow.org/guide/keras/)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zx7wc0LuuxaJ"
      },
      "source": [
        "## Penguin classification problem \n",
        "\n",
        "Imagine you are an ornithologist seeking an automated way to categorize each penguin you find. Machine learning provides many algorithms to classify penguins statistically. For instance, a sophisticated machine learning program could classify penguins based on photographs. The model you build in this tutorial is a little simpler. It classifies penguins based on their body weight, flipper length, and beaks, specifically the length and width measurements of their [culmen](https://en.wikipedia.org/wiki/Beak#Culmen).\n",
        "\n",
        "There are 18 species of penguins, but in this tutorial you will only attempt to classify the following three:\n",
        "\n",
        "* Chinstrap penguins\n",
        "* Gentoo penguins\n",
        "* Adélie penguins\n",
        "\n",
        "<table>\n",
        "  <tr><td>\n",
        "    <img src=\"https://www.tensorflow.org/tutorials/customization/images/penguins_ds_species.png\" alt=\"Illustration of Chinstrap, Gentoo, and Adélie penguins\"\n",
        "     class=\"no-filter\">\n",
        "  </td></tr>\n",
        "  <tr><td align=\"center\">\n",
        "    <b>Figure 1.</b> <a href=\"https://en.wikipedia.org/wiki/Chinstrap_penguin\">Chinstratp</a>, <a href=\"https://en.wikipedia.org/wiki/Gentoo_penguin\">Gentoo</a>, and <a href=\"https://en.wikipedia.org/wiki/Ad%C3%A9lie_penguin\">Adélie</a> penguins (Artwork by @allison_horst, CC BY-SA 2.0).<br/>&nbsp;\n",
        "  </td></tr>\n",
        "</table>\n",
        "\n",
        "Fortunately, a research team has already created and shared a [dataset of 334 penguins](https://allisonhorst.github.io/palmerpenguins/) with body weight, flipper length, beak measurements, and other data. This dataset is also conveniently available as the [penguins](https://www.tensorflow.org/datasets/catalog/penguins) TensorFlow Dataset.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1J3AuPBT9gyR"
      },
      "source": [
        "## Setup\n",
        "\n",
        "Install the `tfds-nightly` package for the penguins dataset. The `tfds-nightly` package is the nightly released version of the TensorFlow Datasets (TFDS). For more information on TFDS, see [TensorFlow Datasets overview](https://www.tensorflow.org/datasets/overview)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4XXWn1eDZmET"
      },
      "outputs": [],
      "source": [
        "!pip install -q tfds-nightly"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DtGeMicKRGzU"
      },
      "source": [
        "Then select **Runtime > Restart Runtime** from the Colab menu to restart the Colab runtime.\n",
        "\n",
        "Do not proceed with the rest of this tutorial without first restarting the runtime."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G9onjGZWZbA-"
      },
      "source": [
        "Import TensorFlow and the other required Python modules. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jElLULrDhQZR"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import tensorflow as tf\n",
        "import tensorflow_datasets as tfds\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "print(\"TensorFlow version: {}\".format(tf.__version__))\n",
        "print(\"TensorFlow Datasets version: \",tfds.__version__)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3Px6KAg0Jowz"
      },
      "source": [
        "## Import the dataset\n",
        "\n",
        "The default [penguins/processed](https://www.tensorflow.org/datasets/catalog/penguins) TensorFlow Dataset is already cleaned, normalized, and ready for building a model. Before you download the processed data, preview a simplified version to get familiar with the original penguin survey data.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qnX1-aLors4S"
      },
      "source": [
        "### Preview the data\n",
        "\n",
        "Download the simplified version of the penguins dataset (`penguins/simple`) using the TensorFlow Datasets [`tdfs.load`](https://www.tensorflow.org/datasets/api_docs/python/tfds/load) method. There are 344 data records in this dataset. Extract the first five records into a [`DataFrame`](https://www.tensorflow.org/datasets/api_docs/python/tfds/as_dataframe) object to inspect a sample of the values in this dataset:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FQvb_JYdrpPm"
      },
      "outputs": [],
      "source": [
        "ds_preview, info = tfds.load('penguins/simple', split='train', with_info=True)\n",
        "df = tfds.as_dataframe(ds_preview.take(5), info)\n",
        "print(df)\n",
        "print(info.features)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kQhzD6P-uBoq"
      },
      "source": [
        "The numbered rows are data records, one _[example](https://developers.google.com/machine-learning/glossary/#example)_ per line, where:\n",
        "  * The first six fields are _[features](https://developers.google.com/machine-learning/glossary/#feature)_: these are the characteristics of an example. Here, the fields hold numbers representing penguin measurements.\n",
        "  * The last column is the _[label](https://developers.google.com/machine-learning/glossary/#label)_: this is the value you want to predict. For this dataset, it's an integer value of 0, 1, or 2 that corresponds to a penguin species name."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CCtwLoJhhDNc"
      },
      "source": [
        "In the dataset, the label for the penguin species is represented as a number to make it easier to work with in the model you are building. These numbers correspond to the following penguin species:\n",
        "\n",
        "* `0`: Adélie penguin\n",
        "* `1`: Chinstrap penguin\n",
        "* `2`: Gentoo penguin\n",
        "\n",
        "Create a list containing the penguin species names in this order. You will use this list to interpret the output of the classification model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sVNlJlUOhkoX"
      },
      "outputs": [],
      "source": [
        "class_names = ['Adélie', 'Chinstrap', 'Gentoo']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iav9kEgxpY0s"
      },
      "source": [
        "For more information about features and labels, refer to the [ML Terminology section of the Machine Learning Crash Course](https://developers.google.com/machine-learning/crash-course/framing/ml-terminology)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PD33PxSmCrtL"
      },
      "source": [
        "### Download the preprocessed dataset\n",
        "\n",
        "Now, download the preprocessed penguins dataset (`penguins/processed`) with the `tfds.load` method, which returns a list of `tf.data.Dataset` objects. Note that the `penguins/processed` dataset doesn't come with its own test set, so use an 80:20 split to [slice the full dataset](https://www.tensorflow.org/datasets/splits) into the training and test sets. You will use the test dataset later to verify your model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EVV96zIYYAi8"
      },
      "outputs": [],
      "source": [
        "ds_split, info = tfds.load(\"penguins/processed\", split=['train[:20%]', 'train[20%:]'], as_supervised=True, with_info=True)\n",
        "\n",
        "ds_test = ds_split[0]\n",
        "ds_train = ds_split[1]\n",
        "assert isinstance(ds_test, tf.data.Dataset)\n",
        "\n",
        "print(info.features)\n",
        "df_test = tfds.as_dataframe(ds_test.take(5), info)\n",
        "print(\"Test dataset sample: \")\n",
        "print(df_test)\n",
        "\n",
        "df_train = tfds.as_dataframe(ds_train.take(5), info)\n",
        "print(\"Train dataset sample: \")\n",
        "print(df_train)\n",
        "\n",
        "ds_train_batch = ds_train.batch(32)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xX2NfLyQOK1y"
      },
      "source": [
        "Notice that this version of the dataset has been processed by reducing the data down to four normalized features and a species label. In this format, the data can be quickly used to train a model without further processing."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iDuG94H-C122"
      },
      "outputs": [],
      "source": [
        "features, labels = next(iter(ds_train_batch))\n",
        "\n",
        "print(features)\n",
        "print(labels)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E63mArnQaAGz"
      },
      "source": [
        "You can visualize some clusters by plotting a few features from the batch:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "me5Wn-9FcyyO"
      },
      "outputs": [],
      "source": [
        "plt.scatter(features[:,0],\n",
        "            features[:,2],\n",
        "            c=labels,\n",
        "            cmap='viridis')\n",
        "\n",
        "plt.xlabel(\"Body Mass\")\n",
        "plt.ylabel(\"Culmen Length\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LsaVrtNM3Tx5"
      },
      "source": [
        "## Build a simple linear model\n",
        "\n",
        "### Why model?\n",
        "\n",
        "A *[model](https://developers.google.com/machine-learning/crash-course/glossary#model)* is a relationship between features and the label.  For the penguin classification problem, the model defines the relationship between the body mass, flipper and culmen measurements and the predicted penguin species. Some simple models can be described with a few lines of algebra, but complex machine learning models have a large number of parameters that are difficult to summarize.\n",
        "\n",
        "Could you determine the relationship between the four features and the penguin species *without* using machine learning?  That is, could you use traditional programming techniques (for example, a lot of conditional statements) to create a model?  Perhaps—if you analyzed the dataset long enough to determine the relationships between body mass and culmen measurements to a particular species. And this becomes difficult—maybe impossible—on more complicated datasets. A good machine learning approach *determines the model for you*. If you feed enough representative examples into the right machine learning model type, the program figures out the relationships for you.\n",
        "\n",
        "### Select the model\n",
        "\n",
        "Next you need to select the kind of model to train. There are many types of models and picking a good one takes experience. This tutorial uses a neural network to solve the penguin classification problem. [*Neural networks*](https://developers.google.com/machine-learning/glossary/#neural_network) can find complex relationships between features and the label. It is a highly-structured graph, organized into one or more [*hidden layers*](https://developers.google.com/machine-learning/glossary/#hidden_layer). Each hidden layer consists of one or more [*neurons*](https://developers.google.com/machine-learning/glossary/#neuron). There are several categories of neural networks and this program uses a dense, or [*fully-connected neural network*](https://developers.google.com/machine-learning/glossary/#fully_connected_layer): the neurons in one layer receive input connections from *every* neuron in the previous layer. For example, Figure 2 illustrates a dense neural network consisting of an input layer, two hidden layers, and an output layer:\n",
        "\n",
        "\n",
        "\n",
        "<table>\n",
        "  <tr><td>\n",
        "    <img src=\"https://www.tensorflow.org/tutorials/customization/images/full_network_penguin.png\" alt=\"A diagram of the network architecture: Inputs, 2 hidden layers, and outputs\"\n",
        "     class=\"no-filter\">\n",
        "  </td></tr>\n",
        "  <tr><td align=\"center\">\n",
        "    <b>Figure 2.</b> A neural network with features, hidden layers, and predictions.<br/>&nbsp;\n",
        "  </td></tr>\n",
        "</table>\n",
        "\n",
        "When you train the model from Figure 2 and feed it an unlabeled example, it yields three predictions: the likelihood that this penguin is the given penguin species. This prediction is called [*inference*](https://developers.google.com/machine-learning/crash-course/glossary#inference). For this example, the sum of the output predictions is 1.0. In Figure 2, this prediction breaks down as: `0.02` for *Adelie*, `0.95` for *Chinstrap*, and `0.03` for *Gentoo* species. This means that the model predicts—with 95% probability—that an unlabeled example penguin is a *Chinstrap* penguin."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W23DIMVPQEBt"
      },
      "source": [
        "### Create a model using Keras\n",
        "\n",
        "The TensorFlow `tf.keras` API is the preferred way to create models and layers. This makes it easy to build models and experiment while Keras handles the complexity of connecting everything together.\n",
        "\n",
        "The `tf.keras.Sequential` model is a linear stack of layers. Its constructor takes a list of layer instances, in this case, two `tf.keras.layers.Dense` layers with 10 nodes each, and an output layer with 3 nodes representing your label predictions. The first layer's `input_shape` parameter corresponds to the number of features from the dataset, and is required:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2fZ6oL2ig3ZK"
      },
      "outputs": [],
      "source": [
        "model = tf.keras.Sequential([\n",
        "  tf.keras.layers.Dense(10, activation=tf.nn.relu, input_shape=(4,)),  # input shape required\n",
        "  tf.keras.layers.Dense(10, activation=tf.nn.relu),\n",
        "  tf.keras.layers.Dense(3)\n",
        "])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FHcbEzMpxbHL"
      },
      "source": [
        "The [*activation function*](https://developers.google.com/machine-learning/crash-course/glossary#activation_function) determines the output shape of each node in the layer. These non-linearities are important—without them the model would be equivalent to a single layer. There are many `tf.keras.activations`, but [ReLU](https://developers.google.com/machine-learning/crash-course/glossary#ReLU) is common for hidden layers.\n",
        "\n",
        "The ideal number of hidden layers and neurons depends on the problem and the dataset. Like many aspects of machine learning, picking the best shape of the neural network requires a mixture of knowledge and experimentation. As a rule of thumb, increasing the number of hidden layers and neurons typically creates a more powerful model, which requires more data to train effectively."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2wFKnhWCpDSS"
      },
      "source": [
        "### Use the model\n",
        "\n",
        "Let's have a quick look at what this model does to a batch of features:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xe6SQ5NrpB-I"
      },
      "outputs": [],
      "source": [
        "predictions = model(features)\n",
        "predictions[:5]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wxyXOhwVr5S3"
      },
      "source": [
        "Here, each example returns a [logit](https://developers.google.com/machine-learning/crash-course/glossary#logits) for each class.\n",
        "\n",
        "To convert these logits to a probability for each class, use the [softmax](https://developers.google.com/machine-learning/crash-course/glossary#softmax) function:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_tRwHZmTNTX2"
      },
      "outputs": [],
      "source": [
        "tf.nn.softmax(predictions[:5])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uRZmchElo481"
      },
      "source": [
        "Taking the `tf.math.argmax` across classes gives us the predicted class index. But, the model hasn't been trained yet, so these aren't good predictions:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-Jzm_GoErz8B"
      },
      "outputs": [],
      "source": [
        "print(\"Prediction: {}\".format(tf.argmax(predictions, axis=1)))\n",
        "print(\"    Labels: {}\".format(labels))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vzq2E5J2QMtw"
      },
      "source": [
        "## Train the model\n",
        "\n",
        "[*Training*](https://developers.google.com/machine-learning/crash-course/glossary#training) is the stage of machine learning when the model is gradually optimized, or the model *learns* the dataset. The goal is to learn enough about the structure of the training dataset to make predictions about unseen data. If you learn *too much* about the training dataset, then the predictions only work for the data it has seen and will not be generalizable. This problem is called [*overfitting*](https://developers.google.com/machine-learning/crash-course/glossary#overfitting)—it's like memorizing the answers instead of understanding how to solve a problem.\n",
        "\n",
        "The penguin classification problem is an example of [*supervised machine learning*](https://developers.google.com/machine-learning/glossary/#supervised_machine_learning): the model is trained from examples that contain labels. In [*unsupervised machine learning*](https://developers.google.com/machine-learning/glossary/#unsupervised_machine_learning), the examples don't contain labels. Instead, the model typically finds patterns among the features."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RaKp8aEjKX6B"
      },
      "source": [
        "### Define the loss and gradients function\n",
        "\n",
        "Both training and evaluation stages need to calculate the model's [*loss*](https://developers.google.com/machine-learning/crash-course/glossary#loss). This measures how off a model's predictions are from the desired label, in other words, how bad the model is performing. You want to minimize, or optimize, this value.\n",
        "\n",
        "Your model will calculate its loss using the `tf.keras.losses.SparseCategoricalCrossentropy` function which takes the model's class probability predictions and the desired label, and returns the average loss across the examples."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QOsi6b-1CXIn"
      },
      "outputs": [],
      "source": [
        "loss_object = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tMAT4DcMPwI-"
      },
      "outputs": [],
      "source": [
        "def loss(model, x, y, training):\n",
        "  # training=training is needed only if there are layers with different\n",
        "  # behavior during training versus inference (e.g. Dropout).\n",
        "  y_ = model(x, training=training)\n",
        "\n",
        "  return loss_object(y_true=y, y_pred=y_)\n",
        "\n",
        "l = loss(model, features, labels, training=False)\n",
        "print(\"Loss test: {}\".format(l))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3IcPqA24QM6B"
      },
      "source": [
        "Use the `tf.GradientTape` context to calculate the [*gradients*](https://developers.google.com/machine-learning/crash-course/glossary#gradient) used to optimize your model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "x57HcKWhKkei"
      },
      "outputs": [],
      "source": [
        "def grad(model, inputs, targets):\n",
        "  with tf.GradientTape() as tape:\n",
        "    loss_value = loss(model, inputs, targets, training=True)\n",
        "  return loss_value, tape.gradient(loss_value, model.trainable_variables)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lOxFimtlKruu"
      },
      "source": [
        "### Create an optimizer\n",
        "\n",
        "An [*optimizer*](https://developers.google.com/machine-learning/crash-course/glossary#optimizer) applies the computed gradients to the model's parameters to minimize the `loss` function. You can think of the loss function as a curved surface (refer to Figure 3) and you want to find its lowest point by walking around. The gradients point in the direction of steepest ascent—so you'll travel the opposite way and move down the hill. By iteratively calculating the loss and gradient for each batch, you'll adjust the model during training. Gradually, the model will find the best combination of weights and bias to minimize the loss. And the lower the loss, the better the model's predictions.\n",
        "\n",
        "<table>\n",
        "  <tr><td>\n",
        "    <img src=\"https://cs231n.github.io/assets/nn3/opt1.gif\" width=\"70%\"\n",
        "         alt=\"Optimization algorithms visualized over time in 3D space.\">\n",
        "  </td></tr>\n",
        "  <tr><td align=\"center\">\n",
        "    <b>Figure 3.</b> Optimization algorithms visualized over time in 3D space.<br/>(Source: <a href=\"http://cs231n.github.io/neural-networks-3/\">Stanford class CS231n</a>, MIT License, Image credit: <a href=\"https://twitter.com/alecrad\">Alec Radford</a>)\n",
        "  </td></tr>\n",
        "</table>\n",
        "\n",
        "TensorFlow has many optimization algorithms available for training. In this tutorial, you will use the `tf.keras.optimizers.SGD` that implements the [*stochastic gradient descent*](https://developers.google.com/machine-learning/crash-course/glossary#gradient_descent) (SGD) algorithm. The `learning_rate` parameter sets the step size to take for each iteration down the hill. This rate is a [*hyperparameter*](https://developers.google.com/machine-learning/glossary/#hyperparameter) that you'll commonly adjust to achieve better results."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XkUd6UiZa_dF"
      },
      "source": [
        "Instantiate the optimizer with a [*learning rate*](https://developers.google.com/machine-learning/glossary#learning-rate) of `0.01`, a scalar value that is multiplied by the gradient at each iteration of the training:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8xxi2NNGKwG_"
      },
      "outputs": [],
      "source": [
        "optimizer = tf.keras.optimizers.SGD(learning_rate=0.01)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pJVRZ0hP52ZB"
      },
      "source": [
        "Then use this object to calculate a single optimization step:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rxRNTFVe56RG"
      },
      "outputs": [],
      "source": [
        "loss_value, grads = grad(model, features, labels)\n",
        "\n",
        "print(\"Step: {}, Initial Loss: {}\".format(optimizer.iterations.numpy(),\n",
        "                                          loss_value.numpy()))\n",
        "\n",
        "optimizer.apply_gradients(zip(grads, model.trainable_variables))\n",
        "\n",
        "print(\"Step: {},         Loss: {}\".format(optimizer.iterations.numpy(),\n",
        "                                          loss(model, features, labels, training=True).numpy()))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7Y2VSELvwAvW"
      },
      "source": [
        "### Training loop\n",
        "\n",
        "With all the pieces in place, the model is ready for training! A training loop feeds the dataset examples into the model to help it make better predictions. The following code block sets up these training steps:\n",
        "\n",
        "1. Iterate each *epoch*. An epoch is one pass through the dataset.\n",
        "2. Within an epoch, iterate over each example in the training `Dataset` grabbing its *features* (`x`) and *label* (`y`).\n",
        "3. Using the example's features, make a prediction and compare it with the label. Measure the inaccuracy of the prediction and use that to calculate the model's loss and gradients.\n",
        "4. Use an `optimizer` to update the model's parameters.\n",
        "5. Keep track of some stats for visualization.\n",
        "6. Repeat for each epoch.\n",
        "\n",
        "The `num_epochs` variable is the number of times to loop over the dataset collection. In the code below, `num_epochs` is set to 201 which means this training loop will run 201 times. Counter-intuitively, training a model longer does not guarantee a better model. `num_epochs` is a [*hyperparameter*](https://developers.google.com/machine-learning/glossary/#hyperparameter) that you can tune. Choosing the right number usually requires both experience and experimentation:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AIgulGRUhpto"
      },
      "outputs": [],
      "source": [
        "## Note: Rerunning this cell uses the same model parameters\n",
        "\n",
        "# Keep results for plotting\n",
        "train_loss_results = []\n",
        "train_accuracy_results = []\n",
        "\n",
        "num_epochs = 201\n",
        "\n",
        "for epoch in range(num_epochs):\n",
        "  epoch_loss_avg = tf.keras.metrics.Mean()\n",
        "  epoch_accuracy = tf.keras.metrics.SparseCategoricalAccuracy()\n",
        "\n",
        "  # Training loop - using batches of 32\n",
        "  for x, y in ds_train_batch:\n",
        "    # Optimize the model\n",
        "    loss_value, grads = grad(model, x, y)\n",
        "    optimizer.apply_gradients(zip(grads, model.trainable_variables))\n",
        "\n",
        "    # Track progress\n",
        "    epoch_loss_avg.update_state(loss_value)  # Add current batch loss\n",
        "    # Compare predicted label to actual label\n",
        "    # training=True is needed only if there are layers with different\n",
        "    # behavior during training versus inference (e.g. Dropout).\n",
        "    epoch_accuracy.update_state(y, model(x, training=True))\n",
        "\n",
        "  # End epoch\n",
        "  train_loss_results.append(epoch_loss_avg.result())\n",
        "  train_accuracy_results.append(epoch_accuracy.result())\n",
        "\n",
        "  if epoch % 50 == 0:\n",
        "    print(\"Epoch {:03d}: Loss: {:.3f}, Accuracy: {:.3%}\".format(epoch,\n",
        "                                                                epoch_loss_avg.result(),\n",
        "                                                                epoch_accuracy.result()))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Diep-ROEuKyl"
      },
      "source": [
        "Alternatively, you could use the built-in Keras [`Model.fit(ds_train_batch)`](https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit) method to train your model. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2FQHVUnm_rjw"
      },
      "source": [
        "### Visualize the loss function over time"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j3wdbmtLVTyr"
      },
      "source": [
        "While it's helpful to print out the model's training progress, you can visualize the progress with [TensorBoard](https://www.tensorflow.org/tensorboard) - a visualization and metrics tool that is packaged with TensorFlow. For this simple example, you will create basic charts using the `matplotlib` module.\n",
        "\n",
        "Interpreting these charts takes some experience, but in general you want to see the *loss* decrease and the *accuracy* increase:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "agjvNd2iUGFn"
      },
      "outputs": [],
      "source": [
        "fig, axes = plt.subplots(2, sharex=True, figsize=(12, 8))\n",
        "fig.suptitle('Training Metrics')\n",
        "\n",
        "axes[0].set_ylabel(\"Loss\", fontsize=14)\n",
        "axes[0].plot(train_loss_results)\n",
        "\n",
        "axes[1].set_ylabel(\"Accuracy\", fontsize=14)\n",
        "axes[1].set_xlabel(\"Epoch\", fontsize=14)\n",
        "axes[1].plot(train_accuracy_results)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zg8GoMZhLpGH"
      },
      "source": [
        "## Evaluate the model's effectiveness\n",
        "\n",
        "Now that the model is trained, you can get some statistics on its performance.\n",
        "\n",
        "*Evaluating* means determining how effectively the model makes predictions. To determine the model's effectiveness at penguin classification, pass some measurements to the model and ask the model to predict what penguin species they represent. Then compare the model's predictions against the actual label.  For example, a model that picked the correct species on half the input examples has an [*accuracy*](https://developers.google.com/machine-learning/glossary/#accuracy) of `0.5`. Figure 4 shows a slightly more effective model, getting 4 out of 5 predictions correct at 80% accuracy:\n",
        "\n",
        "<table cellpadding=\"8\" border=\"0\">\n",
        "  <colgroup>\n",
        "    <col span=\"4\" >\n",
        "    <col span=\"1\" bgcolor=\"lightblue\">\n",
        "    <col span=\"1\" bgcolor=\"lightgreen\">\n",
        "  </colgroup>\n",
        "  <tr bgcolor=\"lightgray\">\n",
        "    <th colspan=\"4\">Example features</th>\n",
        "    <th colspan=\"1\">Label</th>\n",
        "    <th colspan=\"1\" >Model prediction</th>\n",
        "  </tr>\n",
        "  <tr>\n",
        "    <td>5.9</td><td>3.0</td><td>4.3</td><td>1.5</td><td align=\"center\">1</td><td align=\"center\">1</td>\n",
        "  </tr>\n",
        "  <tr>\n",
        "    <td>6.9</td><td>3.1</td><td>5.4</td><td>2.1</td><td align=\"center\">2</td><td align=\"center\">2</td>\n",
        "  </tr>\n",
        "  <tr>\n",
        "    <td>5.1</td><td>3.3</td><td>1.7</td><td>0.5</td><td align=\"center\">0</td><td align=\"center\">0</td>\n",
        "  </tr>\n",
        "  <tr>\n",
        "    <td>6.0</td> <td>3.4</td> <td>4.5</td> <td>1.6</td> <td align=\"center\">1</td><td align=\"center\" bgcolor=\"red\">2</td>\n",
        "  </tr>\n",
        "  <tr>\n",
        "    <td>5.5</td><td>2.5</td><td>4.0</td><td>1.3</td><td align=\"center\">1</td><td align=\"center\">1</td>\n",
        "  </tr>\n",
        "  <tr><td align=\"center\" colspan=\"6\">\n",
        "    <b>Figure 4.</b> A penguin classifier that is 80% accurate.<br/>&nbsp;\n",
        "  </td></tr>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z-EvK7hGL0d8"
      },
      "source": [
        "### Set up the test set\n",
        "\n",
        "Evaluating the model is similar to training the model. The biggest difference is the examples come from a separate *[test set](https://developers.google.com/machine-learning/crash-course/glossary#test_set)* rather than the training set. To fairly assess a model's effectiveness, the examples used to evaluate a model must be different from the examples used to train the model.\n",
        "\n",
        "The penguin dataset doesn't have a separate test dataset so in the previous Download the dataset section, you split the original dataset into test and train datasets. Use the `ds_test_batch` dataset for the evaluation."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HFuOKXJdMAdm"
      },
      "source": [
        "### Evaluate the model on the test dataset\n",
        "\n",
        "Unlike the training stage, the model only evaluates a single [epoch](https://developers.google.com/machine-learning/glossary/#epoch) of the test data. The following code iterates over each example in the test set and compare the model's prediction against the actual label. This comparison is used to measure the model's accuracy across the entire test set:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Tw03-MK1cYId"
      },
      "outputs": [],
      "source": [
        "test_accuracy = tf.keras.metrics.Accuracy()\n",
        "ds_test_batch = ds_test.batch(10)\n",
        "\n",
        "for (x, y) in ds_test_batch:\n",
        "  # training=False is needed only if there are layers with different\n",
        "  # behavior during training versus inference (e.g. Dropout).\n",
        "  logits = model(x, training=False)\n",
        "  prediction = tf.argmax(logits, axis=1, output_type=tf.int64)\n",
        "  test_accuracy(prediction, y)\n",
        "\n",
        "print(\"Test set accuracy: {:.3%}\".format(test_accuracy.result()))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fel8ql2qzGlK"
      },
      "source": [
        "You can also use the `model.evaluate(ds_test, return_dict=True)` keras function to get accuracy information on your test dataset. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HcKEZMtCOeK-"
      },
      "source": [
        "By inspecting the last batch, for example, you can observe that the model predictions are usually correct.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uNwt2eMeOane"
      },
      "outputs": [],
      "source": [
        "tf.stack([y,prediction],axis=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7Li2r1tYvW7S"
      },
      "source": [
        "## Use the trained model to make predictions\n",
        "\n",
        "You've trained a model and \"proven\" that it's good—but not perfect—at classifying penguin species. Now let's use the trained model to make some predictions on [*unlabeled examples*](https://developers.google.com/machine-learning/glossary/#unlabeled_example); that is, on examples that contain features but not labels.\n",
        "\n",
        "In real-life, the unlabeled examples could come from lots of different sources including apps, CSV files, and data feeds. For this tutorial, manually provide three unlabeled examples to predict their labels. Recall, the label numbers are mapped to a named representation as:\n",
        "\n",
        "* `0`: Adélie penguin\n",
        "* `1`: Chinstrap penguin\n",
        "* `2`: Gentoo penguin"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kesTS5Lzv-M2"
      },
      "outputs": [],
      "source": [
        "predict_dataset = tf.convert_to_tensor([\n",
        "    [0.3, 0.8, 0.4, 0.5,],\n",
        "    [0.4, 0.1, 0.8, 0.5,],\n",
        "    [0.7, 0.9, 0.8, 0.4]\n",
        "])\n",
        "\n",
        "# training=False is needed only if there are layers with different\n",
        "# behavior during training versus inference (e.g. Dropout).\n",
        "predictions = model(predict_dataset, training=False)\n",
        "\n",
        "for i, logits in enumerate(predictions):\n",
        "  class_idx = tf.argmax(logits).numpy()\n",
        "  p = tf.nn.softmax(logits)[class_idx]\n",
        "  name = class_names[class_idx]\n",
        "  print(\"Example {} prediction: {} ({:4.1f}%)\".format(i, name, 100*p))"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "custom_training_walkthrough.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
